Random Forest in the Health Industry

Introduction

Machine learning has significantly advanced predictive analytics, particularly in the medical industry and clinical decision-making. Among the many available algorithms, Random Forest (RF) has emerged as a powerful tool due to its ability to handle high-dimensional data, resistance to overfitting, and high accuracy in predicting medical events (Rigatti 2017). RF is an ensemble learning method composed of multiple decision trees, which are generated through bagging and random feature selection. The combined efforts of these trees in bootstrap aggregation allow for superior classification and regression predictions compared to classical statistical models (Biau and Scornet 2016). This flexibility enables biomedical experts to tackle various tasks, including cancer survival analysis, disease progression prediction, and healthcare resource optimization.

One of the most notable applications of RF in healthcare is survival analysis, particularly in colon cancer research. Studies utilizing SEER data have compared RF to the Cox proportional hazards model, highlighting RF’s ability to handle missing data and complex interactions more effectively (Breiman 2001). Additionally, RF has been widely used in clinical decision-making, such as predicting ICU patient outcomes and identifying those at high risk of sepsis.Another critical application is diabetes prediction and prevention. RF not only forecasts diabetes development but also provides personalized recommendations to healthcare professionals, helping them implement preventive measures and improve patient outcomes(Khine and Tun 2022).

Beyond diagnostics, RF is also instrumental in handling imbalanced datasets. For instance, in predicting disease susceptibility, random subsampling techniques within RF have been shown to outperform other machine learning models such as support vector machines, boosting, and bagging (Khalilia, Chakraborty, and Popescu 2011). Furthermore, RF has been used in healthcare resource planning, such as forecasting the demand for essential medications in public health facilities to ensure supply chain efficiency and prevent shortages (Mbonyinshuti, Nshimiyimana, and Uwitonze 2022).

RF is a powerful algorithm, but it also has its challenges. Hyperparameter tuning is crucial for maximizing predictive accuracy, with factors such as the number of trees (L), sample size per tree, and the number of variables considered at each split (mtry) all significantly influencing model performance (Probst, Wright, and Boulesteix 2019b). While RF often performs well with default settings, fine-tuning these parameters can enhance both reliability and speed (Boulesteix et al. 2012). However, the major drawback remains model interpretability, which is critical in medical decision-making. Researchers have proposed methods such as conditional inference forests (CIF) to improve reliability while reducing bias in variable selection (Dai et al. 2018).

Since RF was first introduced to clinical diagnostics, it has been regarded as one of the most effective tools in medical prediction models. This study explores its performance compared to traditional statistical techniques and evaluates whether advancements in hyperparameter optimization can further enhance its effectiveness. By synthesizing insights from multiple sources, this analysis provides a comprehensive understanding of RF’s potential in healthcare analytics while identifying areas for further refinement.

Methods

The random forest algorithm generates numerous decision trees using randomization and then aggregates the output of these trees into one output. A decision tree is an algorithm that has a ‘tree-like’ structure, consisting of a root node, branches, internal nodes and leaf nodes. The root node branches out into internal nodes, and depending on the outcome of each internal node, it ultimately leads to the leaf node; or the final outcome, or decision. In the formation of a random forest, a voting method is used for classification and an averaging method for regression. Randomization is done in two steps. The first uses bootstrap aggregation or bagging at the data set level, creating new randomized samples for the model development and testing. Bagging consists of randomly sampling from the original data set with replacement, ensuring the sample set is completely random. The data that is not used in the sampled data set is considered the out-of-bag (OOB) data. The next level of randomization happens at the decision node level. A certain number of predictors are chosen, which is often the square root of the number of predictors in the data set. The algorithm tests all possible thresholds for all selected variables and chooses the variable-threshold combination which results in the best split – the split which most effectively separates cases from controls, for instance (Rigatti 2017). This random selection of variables and threshold testing continues until either “pure” nodes are reached (containing only cases or controls) or some pre-defined endpoint (Rigatti 2017).

Hyper parameters Overview

Table 1 below shows a summary of the different hyperparameters of random forest and typical default values. Here, n is the number of observations and p is the number of variables in the dataset.

Table 1. (Probst, Wright, and Boulesteix 2019b)

Table 1 provides an overview of the hyperparameters available to tune the decision forest, along with the recommended or default values. The decision forest algorithm is designed to work well without much tuning. It’s been found that an increased number of trees provides a better output, but there’s typically a limit in which there’s no more gain in accuracy, and only a decrease or slow down in performance with too many trees. The hyperparameter ‘mtry’ has been found to have the most influence, where the best value of ‘mtry’ depends on the number of variables that are related to the outcome (Probst, Wright, and Boulesteix 2019b).

Gini Impurity

The random forest algorithm uses the Gini measure of impurity to select the split with the lowest impurity at every node (Khalilia, Chakraborty, and Popescu 2011). The gini impurity can range between 0 - 0.5, with the lower the impurity, the better the split. If a gini impurity is 0, this means it is a ‘pure’ node and does not need to be split further. The formula for Gini Impurity is below, where p is the probability of samples belonging to the class i at a specific node.

\[ Gini = 1 -\sum_{i = 1}^{n}{(p_i)^2} \]

Random Forest Implementation in the Dataset

The random forest model will be applied to analyze the dataset containing multiple clinical and demographic factors. This dataset has features like Age, Sex, Type of Chest Pain, Resting Blood Pressure (RestingBP), Cholesterol level, Fasting Blood Sugar (FastingBS), Electrocardiogram Results when at Rest (RestingECG), Maximum Heart Rate attained (MaxHR), Angina induced by exercise, Oldpeak, and ST Slope. The model will be fitted to the data so the presence or lack of heart disease (HeartDisease) can be predicted. This variable is the target (0 - No Heart Disease, 1 - Heart Disease).

Model Training

In order to train the model, a dataset will first be created by splitting it into an 70% training and 30% testing set for evaluation. The random forest will have an ensemble of 100 decision trees (n_estimators = 100) and the mtry value for each split will be the square root of the total number of features (Oshiro, Perez, and Baranauskas 2012). The Gini impurity metric will be utilized and will ensure that the most informative features are selected (Khalilia, Chakraborty, and Popescu 2011). Other parameter control methodologies such as grid or randomized search will be used to optimize the number of trees, maximum depth, and minimum sample split (Probst, Wright, and Boulesteix 2019b).

Model Evaluation and Performance Metrics

For the model evaluation, the following metrics will be applied:

• AUC-ROC Score: Evaluates the model’s classification capability for distinguishing classes (Heart Disease vs. No Heart Disease) (Probst, Wright, and Boulesteix 2019a). The AUC ROC Score is the area under the curve of the ROC (receiver-operating characteristic curve) curve. The ROC curve graphs the true positive rate over the false positive rate. The higher the ROC AUC, the better. \[ \text{AUC-ROC} = \int_{0}^{1} \text{TPR} \left( \text{FPR}^{-1}(x) \right) \, dx \]

• Precision: Precision is the ratio of the true positive predictions to the total number of positive predictions \[ Precision = \frac{True Positives}{(True Positives + False Positives)} \]

• Recall (Sensitivity): Recall is the ratio of the true positive predictions to all the actual positive instances: \[\text{Recall} = \frac{TruePositives}{TruePositives + FalseNegatives}\]

• F1-score: F1 score is the average of precision and recall. \[ F1 = \frac{2*(Precision*Recall)}{(Precision + Recall)} \]

Advantages/Disadvantages

The random forests method remains one of the most robust and versatile methods for solving classification tasks, especially in the healthcare sector. Their capability to manage high dimensional data, model intricate relationships, and rank order features makes them particularly useful in disease prediction and risk factor evaluation (Breiman 2001). All of these benefits come with caveats. While random forests are robust in overfitting and are efficient with missing values (Khalilia, Chakraborty, and Popescu 2011), they also demand above-average computing power, can be inefficient with time on large databases, and are less clear than logistic regression models (Probst, Wright, and Boulesteix 2019a). Imposing certain values to parameters improves these shortcomings but not without a loss of performance and efficiency in specific datasets. Regardless, random forests are still a good choice in predictive modeling in medical research and decision support systems (Biau and Scornet 2016).

In summary the Random Forest algorithm has been used in this study because of its compatibility with structured clinical datasets and its ability to integrate categorical and numerical data without significant preprocessing (Breiman 2001). In contrast to logistic regression, which applies a mode of linearity, Random Forest is able to capture complex non-linear relationships in healthcare data (Probst, Wright, and Boulesteix 2019a). Another advantage of Random Forest is its provision of feature importance indices, which enhances the model’s interpretability by medical practitioners concerning risk prediction and proactive measures’ implementation(Khalilia, Chakraborty, and Popescu 2011)

Distribution of Features

Figure 1: Distribution of some Features

# The distribution of 'Age' with a histogram - normal distribution
ageplot <- ggplot(data, aes(x = Age)) +
  geom_histogram(bins = 30, fill = "blue", color = "black") +
  theme_minimal() +
  theme(plot.title = element_text(hjust = 0.5)) +
  labs(title = "Age Distribution", x = "Age", y = "Count")

# The distribution of 'Heart Disease' with a histogram - no class imbalance
hdplot <- ggplot(data, aes(x = HeartDisease)) +
  geom_bar(fill = "blue", color = "black") +
  theme_minimal() +
  theme(plot.title = element_text(hjust = 0.5)) +
  scale_x_continuous(breaks=c(0,1)) +
  labs(title = "Heart Disease Class Distribution", x = "Heart Disease", y = "Count")

# The distribution of 'Sex' with a histogram - imbalance: ~4x more males than females
splot <- ggplot(data, aes(x = Sex)) +
  geom_bar(fill = "blue", color = "black") +
  theme_minimal() +
  theme(plot.title = element_text(hjust = 0.5)) +
  labs(title = "Sex Distribution", x = "Sex", y = "Count")

# The distribution of 'Cholesterol' with a histogram - over 150 records with a cholesterol of 0; otherwise normal distribution
cplot <- ggplot(data, aes(x = Cholesterol)) +
  geom_histogram(fill = "blue", color = "black") +
  theme_minimal() +
  theme(plot.title = element_text(hjust = 0.5)) +
  labs(title = "Cholesterol Distribution", x = "Cholesterol", y = "Count")

grid.arrange(hdplot, splot, ageplot, cplot, ncol = 2, nrow=2)

• Distribution/trends which are relevant in predicting heart diseases.

Distribution Explanations

• Heart Disease Distribution (Top-Left): The distribution is appropriately balanced, minimizing the chances of bias in the model.
• Sex Distribution (Top-Right): The dataset has more male patients than female, which might impact predictions.
• Age Distribution (Bottom-Left): Most patients fall within the 40-70 years age range, which may reduce model accuracy for younger individuals.
• Cholesterol Distribution (Bottom-Right): The presence of zero values in cholesterol is unrealistic, indicating the need for data cleaning.

Correlation Matrix

Figure 2: Correlation Matrix – Understanding Key Relationships

# correlation matrix for numeric features
library(corrplot)
numeric_data <- data %>% select(where(is.numeric))
cor_matrix <- cor(numeric_data)
corrplot(cor_matrix, method = "circle", type = "upper", 
         tl.col = "black", tl.cex = 0.7, addCoef.col = "black")

Correlation Matrix Insights

• Figure 2 shows correlations of features in the data set with heart disease.
• Oldpeak (0.40) stands out since higher ST depression is associated with heart disease risk.
• Patients who have heart disease are more likely to have lower maximum heart rate (MaxHR (-0.40)).
• Age (0.28) and Fasting Blood Sugar (0.27) also emerged as positive correlates, confirming that older people and people with high fasting blood sugar levels are at risk.

Modeling and Results

• Begin with performing any necessary cleaning and preprocessing of the data.

• We will then use the decision tree algorithm to demonstrate how a decision tree works, and show the performance of one tree.

• The next step will be using the random forest algorithm, which is a combination of decision trees, to see how it performs in comparison, ideally providing a more accurate prediction.

Data Preprocessing and Cleaning

• No null or NA missing values found in the data set
• One row with a RestingBP = 0
• 172 rows with Cholesterol = 0
• We decided to drop these rows from the data set, as they were missing valid data.

Data Encoding

• The next step is to encode the data.
• The random forest alogorithm, along with most machine learning algorithms, functions best with numerical values.
• Use one-hot encoding to transform the categorical variables into a binary column that indicates the presence (1) or absence (0) of the category.

Encoded Data Preview

Splitting Data

• We split the data set into training and test subsets.

• The training subset will contain 70% of the data.

• The test subset will contain 30% of the data.